184 research outputs found
Approximating the Termination Value of One-Counter MDPs and Stochastic Games
One-counter MDPs (OC-MDPs) and one-counter simple stochastic games (OC-SSGs)
are 1-player, and 2-player turn-based zero-sum, stochastic games played on the
transition graph of classic one-counter automata (equivalently, pushdown
automata with a 1-letter stack alphabet). A key objective for the analysis and
verification of these games is the termination objective, where the players aim
to maximize (minimize, respectively) the probability of hitting counter value
0, starting at a given control state and given counter value. Recently, we
studied qualitative decision problems ("is the optimal termination value = 1?")
for OC-MDPs (and OC-SSGs) and showed them to be decidable in P-time (in NP and
coNP, respectively). However, quantitative decision and approximation problems
("is the optimal termination value ? p", or "approximate the termination value
within epsilon") are far more challenging. This is so in part because optimal
strategies may not exist, and because even when they do exist they can have a
highly non-trivial structure. It thus remained open even whether any of these
quantitative termination problems are computable. In this paper we show that
all quantitative approximation problems for the termination value for OC-MDPs
and OC-SSGs are computable. Specifically, given a OC-SSG, and given epsilon >
0, we can compute a value v that approximates the value of the OC-SSG
termination game within additive error epsilon, and furthermore we can compute
epsilon-optimal strategies for both players in the game. A key ingredient in
our proofs is a subtle martingale, derived from solving certain LPs that we can
associate with a maximizing OC-MDP. An application of Azuma's inequality on
these martingales yields a computable bound for the "wealth" at which a "rich
person's strategy" becomes epsilon-optimal for OC-MDPs.Comment: 35 pages, 1 figure, full version of a paper presented at ICALP 2011,
invited for submission to Information and Computatio
Qualitative Multi-Objective Reachability for Ordered Branching MDPs
We study qualitative multi-objective reachability problems for Ordered
Branching Markov Decision Processes (OBMDPs), or equivalently context-free
MDPs, building on prior results for single-target reachability on Branching
Markov Decision Processes (BMDPs).
We provide two separate algorithms for "almost-sure" and "limit-sure"
multi-target reachability for OBMDPs. Specifically, given an OBMDP,
, given a starting non-terminal, and given a set of target
non-terminals of size , our first algorithm decides whether the
supremum probability, of generating a tree that contains every target
non-terminal in set , is . Our second algorithm decides whether there is
a strategy for the player to almost-surely (with probability ) generate a
tree that contains every target non-terminal in set .
The two separate algorithms are needed: we show that indeed, in this context,
"almost-sure" "limit-sure" for multi-target reachability, meaning that
there are OBMDPs for which the player may not have any strategy to achieve
probability exactly of reaching all targets in set in the same
generated tree, but may have a sequence of strategies that achieve probability
arbitrarily close to . Both algorithms run in time , where is the total bit encoding length
of the given OBMDP, . Hence they run in polynomial time when
is fixed, and are fixed-parameter tractable with respect to . Moreover, we
show that even the qualitative almost-sure (and limit-sure) multi-target
reachability decision problem is in general NP-hard, when the size of the
set of target non-terminals is not fixed.Comment: 47 page
Polynomial Time Algorithms for Branching Markov Decision Processes and Probabilistic Min(Max) Polynomial Bellman Equations
We show that one can approximate the least fixed point solution for a
multivariate system of monotone probabilistic max(min) polynomial equations,
referred to as maxPPSs (and minPPSs, respectively), in time polynomial in both
the encoding size of the system of equations and in log(1/epsilon), where
epsilon > 0 is the desired additive error bound of the solution. (The model of
computation is the standard Turing machine model.) We establish this result
using a generalization of Newton's method which applies to maxPPSs and minPPSs,
even though the underlying functions are only piecewise-differentiable. This
generalizes our recent work which provided a P-time algorithm for purely
probabilistic PPSs.
These equations form the Bellman optimality equations for several important
classes of infinite-state Markov Decision Processes (MDPs). Thus, as a
corollary, we obtain the first polynomial time algorithms for computing to
within arbitrary desired precision the optimal value vector for several classes
of infinite-state MDPs which arise as extensions of classic, and heavily
studied, purely stochastic processes. These include both the problem of
maximizing and mininizing the termination (extinction) probability of
multi-type branching MDPs, stochastic context-free MDPs, and 1-exit Recursive
MDPs.
Furthermore, we also show that we can compute in P-time an epsilon-optimal
policy for both maximizing and minimizing branching, context-free, and
1-exit-Recursive MDPs, for any given desired epsilon > 0. This is despite the
fact that actually computing optimal strategies is Sqrt-Sum-hard and
PosSLP-hard in this setting.
We also derive, as an easy consequence of these results, an FNP upper bound
on the complexity of computing the value (within arbitrary desired precision)
of branching simple stochastic games (BSSGs)
Quasi-Birth-Death Processes, Tree-Like QBDs, Probabilistic 1-Counter Automata, and Pushdown Systems
We begin by observing that (discrete-time) Quasi-Birth-Death Processes (QBDs) are equivalent, in a precise
sense, to probabilistic 1-Counter Automata (p1CAs), and both Tree-Like QBDs (TL-QBDs) and Tree-Structured QBDs (TS-QBDs) are equivalent to both probabilistic Pushdown Systems (pPDSs) and Recursive Markov Chains (RMCs).
We then proceed to exploit these connections to obtain a number of new algorithmic upper and lower bounds for central computational problems about these models. Our main result is this: for an arbitrary QBD, we can approximate its termination probabilities (i.e., its matrix) to within bits of precision (i.e., within additive error ), in time polynomial in \underline{both} the encoding size of the QBD and in , in the unit-cost rational arithmetic RAM model of computation. Specifically,
we show that a decomposed Newton's method can be used to achieve this. We emphasize that this bound is very different from the well-known ``linear/quadratic convergence'' of numerical analysis, known for QBDs and TL-QBDs, which typically gives no constructive bound in terms of the encoding size of the system being solved.
In fact, we observe (based on recent results) that for the more general TL-QBDs such a polynomial upper bound on Newton's method fails badly. Our upper bound proof for QBDs combines several ingredients: a detailed analysis of the structure of 1-counter automata, an iterative application of a classic condition number bound for errors in linear systems, and a very recent constructive bound on the performance of Newton's method for strongly connected monotone systems of polynomial equations.
We show that the quantitative termination decision problem for QBDs (namely, ``is ?'') is at least as hard as long standing open problems in the complexity of exact numerical computation, specifically the square-root sum problem. On the other hand, it follows from our earlier results for RMCs that any non-trivial approximation of termination probabilities for TL-QBDs is sqrt-root-sum-hard
Query Complexity of Approximate Equilibria in Anonymous Games
We study the computation of equilibria of anonymous games, via algorithms
that may proceed via a sequence of adaptive queries to the game's payoff
function, assumed to be unknown initially. The general topic we consider is
\emph{query complexity}, that is, how many queries are necessary or sufficient
to compute an exact or approximate Nash equilibrium.
We show that exact equilibria cannot be found via query-efficient algorithms.
We also give an example of a 2-strategy, 3-player anonymous game that does not
have any exact Nash equilibrium in rational numbers. However, more positive
query-complexity bounds are attainable if either further symmetries of the
utility functions are assumed or we focus on approximate equilibria. We
investigate four sub-classes of anonymous games previously considered by
\cite{bfh09, dp14}.
Our main result is a new randomized query-efficient algorithm that finds a
-approximate Nash equilibrium querying
payoffs and runs in time . This improves on the running
time of pre-existing algorithms for approximate equilibria of anonymous games,
and is the first one to obtain an inverse polynomial approximation in
poly-time. We also show how this can be utilized as an efficient
polynomial-time approximation scheme (PTAS). Furthermore, we prove that
payoffs must be queried in order to find any
-well-supported Nash equilibrium, even by randomized algorithms
Two Variable vs. Linear Temporal Logic in Model Checking and Games
Model checking linear-time properties expressed in first-order logic has
non-elementary complexity, and thus various restricted logical languages are
employed. In this paper we consider two such restricted specification logics,
linear temporal logic (LTL) and two-variable first-order logic (FO2). LTL is
more expressive but FO2 can be more succinct, and hence it is not clear which
should be easier to verify. We take a comprehensive look at the issue, giving a
comparison of verification problems for FO2, LTL, and various sublogics thereof
across a wide range of models. In particular, we look at unary temporal logic
(UTL), a subset of LTL that is expressively equivalent to FO2; we also consider
the stutter-free fragment of FO2, obtained by omitting the successor relation,
and the expressively equivalent fragment of UTL, obtained by omitting the next
and previous connectives. We give three logic-to-automata translations which
can be used to give upper bounds for FO2 and UTL and various sublogics. We
apply these to get new bounds for both non-deterministic systems (hierarchical
and recursive state machines, games) and for probabilistic systems (Markov
chains, recursive Markov chains, and Markov decision processes). We couple
these with matching lower-bound arguments. Next, we look at combining FO2
verification techniques with those for LTL. We present here a language that
subsumes both FO2 and LTL, and inherits the model checking properties of both
languages. Our results give both a unified approach to understanding the
behaviour of FO2 and LTL, along with a nearly comprehensive picture of the
complexity of verification for these logics and their sublogics.Comment: 37 pages, to be published in Logical Methods in Computer Science
journal, includes material presented in Concur 2011 and QEST 2012 extended
abstract
Analyzing probabilistic pushdown automata
The paper gives a summary of the existing results about algorithmic analysis of probabilistic pushdown automata and their subclasses.V článku je podán přehled známých výsledků o pravděpodobnostních zásobníkových automatech a některých jejich podtřídách
Decidability Results for Multi-objective Stochastic Games
We study stochastic two-player turn-based games in which the objective of one
player is to ensure several infinite-horizon total reward objectives, while the
other player attempts to spoil at least one of the objectives. The games have
previously been shown not to be determined, and an approximation algorithm for
computing a Pareto curve has been given. The major drawback of the existing
algorithm is that it needs to compute Pareto curves for finite horizon
objectives (for increasing length of the horizon), and the size of these Pareto
curves can grow unboundedly, even when the infinite-horizon Pareto curve is
small. By adapting existing results, we first give an algorithm that computes
the Pareto curve for determined games. Then, as the main result of the paper,
we show that for the natural class of stopping games and when there are two
reward objectives, the problem of deciding whether a player can ensure
satisfaction of the objectives with given thresholds is decidable. The result
relies on intricate and novel proof which shows that the Pareto curves contain
only finitely many points. As a consequence, we get that the two-objective
discounted-reward problem for unrestricted class of stochastic games is
decidable.Comment: 35 page
Quantitative multi-objective verification for probabilistic systems
We present a verification framework for analysing multiple quantitative objectives of systems that exhibit both nondeterministic and stochastic behaviour. These systems are modelled as probabilistic automata, enriched with cost or reward structures that capture, for example, energy usage or performance metrics. Quantitative properties of these models are expressed in a specification language that incorporates probabilistic safety and liveness properties, expected total cost or reward, and supports multiple objectives of these types. We propose and implement an efficient verification framework for such properties and then present two distinct applications of it: firstly, controller synthesis subject to multiple quantitative objectives; and, secondly, quantitative compositional verification. The practical applicability of both approaches is illustrated with experimental results from several large case studies
Probably Safe or Live
This paper presents a formal characterisation of safety and liveness
properties \`a la Alpern and Schneider for fully probabilistic systems. As for
the classical setting, it is established that any (probabilistic tree) property
is equivalent to a conjunction of a safety and liveness property. A simple
algorithm is provided to obtain such property decomposition for flat
probabilistic CTL (PCTL). A safe fragment of PCTL is identified that provides a
sound and complete characterisation of safety properties. For liveness
properties, we provide two PCTL fragments, a sound and a complete one. We show
that safety properties only have finite counterexamples, whereas liveness
properties have none. We compare our characterisation for qualitative
properties with the one for branching time properties by Manolios and Trefler,
and present sound and complete PCTL fragments for characterising the notions of
strong safety and absolute liveness coined by Sistla
- …